arXiv:1508.02679v2 [cs.NI] 6 Jan 2017 


Network Issues in Virtual Machine Migration 

Hatem Ibn-Khedher*, Emad Abd-Elrahman*, Hossam Afifi* and Jacky Eorestier^^ 

*Institut Mines-Telecom (IMT), Telecom SudParis, Saclay, France. 

Email: {hatem.ibn_khedher, emad.abd_elrahman, hossam.afifi}@telecom-sudparis.eu 
Orange Labs, Issy Les Moulineaux, France. 

Email: J acky. fore slier @ orange. com 


Abstract —Software Defined Networking (SDN) is based ba¬ 
sically on three features: centralization of the control plane, 
programmability of network functions and traffic engineering. 
The network function migration poses interesting problems that 
we try to expose and solve in this paper. Content Distribution 
Network virtualization is presented as use case. 

Index Terms —Virtualization, SDN, NFV, QoS, Mobility 

I. Introduction 

The virtualization of resources has addressed the network 
architecture as a potential target. The basic tasks required in 
the virtualization substrate are instantiation of new network 
functions, migration and switching. These basic tasks are 
strongly dependent on the underlying network configuration 
and topology in a way that makes them tributary of the network 
conditions. It means that sometimes it will not be possible or 
not recommended to accomplish some virtualization tasks if 
the network is not presenting the minimum requirements. This 
brings back many dj vu questions to networks theory but the 
answers to these questions require an understanding of the new 
context. 

Most of the virtualization architectures such as NFV in¬ 
cluded in the SDN paradigm are relying on these tools to 
implement their technical solutions. 

We consider here the migration of network functions and 
we study problems that arise from this dynamic process. 
We present a use case for content distribution of multimedia 
services. 

The rest of this paper is organized as follows; Section 
II highlights the common architectures for network resource 
virtualization. Section III describes the infiuence of the main 
network parameters in the virtualization process. We study 
network basics such as addressing, quality of service and 
mobility issues. Simple improvements and technical solutions 
are presented and demonstrated in Section IV. Section V 
introduced our virtual CDN (vCDN) use case. Finally, this 
work is concluded in Section VI. 

II. NFV, SDN AND OpenStack 

We investigate the state of the art of the networking tech¬ 
nologies that can meet the virtualization domain. The main 
architectures used for that purpose are Network Functions 
Virtualization (NFV), Software Defined Networking (SDN) and 
OpenStack. 


A. Network Functions Virtualization 

Network Functions Virtualization (NFV) |[T| virtualizes the 
network equipment (Router, DPI, Firewall...). We will not 
discuss about hardware. We will rather consider software based 
NFV architecture. It is a concept that decouples network 
functions from its underlying hardware. Then, it enables the 
software to run on virtualized generic environment. Therefore, 
several virtual appliances can share the single hardware re¬ 
sources. 

NFV brings several benefits Q such as reducing CAPEX 
and OPEX, promoting flexibility and innovation of the virtual 
network functions already implemented. Moreover, it has been 
introduced as a new networking facility that poised to amend 
the core structure of telecommunication infrastructure to be 
more cost efficient. 

In a virtualized environment, the physical resources (CPU, 
Storage, RAM) are emulated and the guest operating system 
runs over the emulated hardware (vCPU, vRAM, vStorage 
) as if it is running on its own physical resources. By this 
virtualization, all the virtual machines share simultaneously the 
available resources. 

This virtualization is based on deterministic framework 
which is constituted of three main components: 

• Host machine: It is the physical machine (physical server) 
that owns the hardwares resources (CPU, RAM, Storage, 
Input output interfaces, etc.) 

• Hypervisor: It is the controller of the virtual machines that 
controls their instantiation, dynamic migration, etc. 

• Guest machine: It is the virtual machine or the virtual 
appliance that is running and attached to the hypervisor. 

The standardization of NFV began with ETSI and some use 
cases described in The main use cases found are: 

• Virtualization of the Radio Network Interface: In this 
approach we separate the digital function of the radio 
named Base Band Unit (BBU) from the underlying an¬ 
tenna hardware and distribute the Remote Radio Unit 
(RRH). The virtualization can be done in a data center 
that communicates with the RRH distributed functions 
through optical back-haul networks (optical fiber) in order 
to accelerate the resource allocation. Bhaumik et al. 0 
gave an important result utilizing cloud RAN in which 
they virtualized the Base Band Units and distribute RRU 
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Figure 1: Evolution of multimedia application delivery 


units. They reported that Cloud RAN can decrease the 
network load by 22%. 

• Virtualization of the Home Network: The virtualization 
of the home network includes the virtualization of the 
two main components: Residential Gateway and Set-Top 
Boxes that offer home services (internet access, multi- 
media service, etc.) to end users. This approach is based 
on implementing virtualized and programmable software 
based NFV solution such as: firewalls, DHCP servers, 
VPN Gateways, DPI Gateways. Then, move them to 
data centers in order to decrease the cost of devices and 
increase the QoS. 

• Virtualization of Evolved Packed Core (EPC): In this use 
case, the virtualization targeted several functions such as: 
SGW, PGW, MME, HSS, and PCRE (TJ. The virtual EPC 
will include all the above functions as software based NEV 
solutions moved into a cloud EPC. Using this approach 
we can reduce the network control traffic by 70 percent 
as described in |[T|. 

B. Software Defined Networking 

SDN Q is a networking concept that shares the same goals 
of NEV in the direction of promoting innovation; openness 
and flexibility. We emphasize that they are independent but 
complementary as explained in j^. 

It is characterized by the separation of control plane and 
data plane and consolidates the control plane in one logical 
centralized controller to decrease the network overload and 
increase enhancement the traffic engineering by adding policy 
and rules to devices enabled OpenElow protocol. 

It can be considered as a tool for traffic engineering to 
enhance the QoS and meet network issues following the in¬ 
tegration of virtualization functions. SDN based on OpenElow 
protocol can be used to direct the forwarding layer formed by 
Open Elow enabled devices such as OpenElow Switches. 

As shown in Eig. |T| the multimedia service delivery has 
evolved three times: 

• Phase I: The operators adopt the Best Effort (BE) in ser¬ 
vice delivery from content providers to clients according 
to the available bandwidth and without any regards to 


delay issues, users satisfaction or perception for QoE (i.e. 
no control for QoS). 

• Phase II: The operators use the CDN solution to cache 
the multimedia very near to the clients by using CDN 
network acceleration. In this phase, there is a remarkable 
enhancement in service delivery while still some restric¬ 
tions remain between customers and content providers 
[15]. 

• Phase III: The operators aim to redirect the required ser¬ 
vices from content providers to selected network locations 
giving to clients good perception in QoS or QoE measures. 
In this phase, there is an application layer tunneling 
defined by SDN (QoS/QoE perception model) [16]. 

Our concern in this paper is to benefit from these comple¬ 
mentarities in order to virtualize and migrate the multimedia 
network functions across hosts. Moreover, we aim to gain 
the advantages of scalability and cost reduction in a dynamic 
controlled service delivery context. We focus in this work on 
the mobility issues that arise when achieving the multimedia 
network function migration. 

Mechanism of SDN in Video Service Delivery 

SDN will lead to Software Defined Data Centers (SDDC) 
where the roots of streaming points dynamically move. This 
dynamic adaptation will pass through 4 phases as follows: 

• Resources Virtualization: It concerns the different re¬ 
sources needed during virtualization including bandwidth, 
system (memory, CPU, I/O). 

• Roots & Links Virtualization: The virtual resources gath¬ 
ered and calculated in real time for the virtual node 
inserted among different nodes in the original tree of 
operators networks. 

• Network Virtualization: It is a kind of dynamic networks 
or nodes on demand to serve as streaming points. 

• Elows Virtualization: It is a mapping of original root fiows 
to the new elected roots through SDN tunnels between the 
old root and virtual root to new ones. 

C. Openstack 

OpenStack [7] is a collection of services and software. It 
is an open source project that replaces the Infrastructure as a 
Service (laaS) layer in cloud computing domain. It can deal 
with the two networking technologies that appeared: NEV and 
SDN. Its conceptual architecture is showed in Eig. 

However the main components of OpenStack are: Nova, 
Glance and Swift. Glance service is the creator of image disks 
and Swift project is the manager of the storage in the private 
cloud (the equivalent of S3 service in Amazon EC2 project). 

OpenStack has also an orchestrator module (Heat) that 
manages the virtual network functions (VNEs) in NEV based 
Eramework. Their components are well cooperating so as to 
achieve dynamic instantiation of VMs in the container (VNE) 
through Nova module which is the most important service in 
this project. It included also Keystone which the Manager of 
the identity. 
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Figure 2: OpenStack architecture [7] 


III. Network Constraints in Virtualization 

We study mobility in the virtualization process. Several basic 
requirements are necessary before any virtualization procedure 
can take place. We think that QoS, mobility support and 
security are among the most important issues to be addressed: 

A. QoS 

This parameter is vital. Any virtualization process consumes 
in its transient state a huge amount of resources. Instantiation 
or migration of a VM requires very high speed links. Slow 
links do not only render the virtualization very hazardous but 
can simply make it impossible to accomplish. QoS is required 
in many situations. It is to be mentioned that virtual machine 
migration as of today is not optimized and very trivial approach 
where everything is copied from the origin to the destination. 
Hence, we can expect up to 90 per cent of the copied material is 
un-necessary because it is unchanged. We expect that efforts in 
live migration will be done to optimize what is being copied. 
The QoS of the link can hence be a decisive constraint on 
whether to migrate or not. 

B. Mobility 

Mobility of virtual instances is not a simple task. We can 
consider moving an NF or a complete instance of a server 
depending on the desired controller objective. So, from the 
operating system perspective, there will be a virtual machine, 
a process, a thread or a session migration as a result of this mo¬ 
bility. As explained before, moving a functionality is required 
when we want to create a new service in a different location. 
Off-loading, QoS improvement, interface management [18] etc. 
may be the reasons for this mobility. When we consider moving 
a complete virtual machine, it is mainly for load balancing 


but can also have other reasons such firmware/OS upgrade, 
instantiation of a new service for a new customer, etc. 

C. Security 

Security is an important aspect in VMs migrations. Many 
attacks could stop the live or offloading migration at any point. 
So, securing this migration either in single domain or inter 
multiple domains is mandatory. Tools such as Openstack do not 
authorize all the possible operations when different domains are 
involved. This could present a limitation in real deployment. 

IV. Networks Issues for Virtualized Network 
Functions Mobility 

Virtualized network functions need for an added network 
requirement in order to assure session moving, open session 
and live session migration. For basic knowledge, if a virtual 
machine that resides on a given network, obviously executed as 
a process, changes the network; it suffers from the continuity 
of the sessions offered to users which have connected to it. 
This of course is due to change of VMs IP. Therefore, we 
must address this issue to ensure the continuity of the session 
after the migration of the virtual appliance to a remote host 
and a remote Hypervisor. 

A. Hypervisor Overview 

Hypervisor is the virtual machine monitor that controls 
virtual machine issue and specially network issues of VM 
and during the live migration process. It allows to move vir¬ 
tual machines between different domains. Among the famous 
hypervisors, we cite VMware ||^, Xen |[^, Qemu [10], and 
VirtualBox [11] etc. 

In our paper, we use two tools. Qemu is first tested as an 
open source hypervisor that can monitor the running virtual 
machine and deal with virtualization tools that cant be used in 
another virtualized environment such as: KVM. We developped 
also a special tool for Optimized migration based on Openstack 
(D based on linear optimization tools fT^ and p0| 

The aforementioned hypervisors enabled VM live migration 
with continuity of the running service. However, they missed 
several requirements for assuring live migration and open 
session. Those requirements are mainly concerning IP mobility 
when VMs migrate from source host to destination one as it 
is explained in [12] and [13]. 

Q. Li at.al [14] introduced that mobility issue can be 
controlled either by the VM itself, by the host machine (or 
any intermediate network node) or by the hypervisor. We will 
focus on those ways of enabling and controlling IP mobility 
when VMs migrate from one host to another and prove that 
the best way to control IP mobility is by the hypervisor. 

In [17] Kalim et al. introduced another approach to migrate 
virtual machines between different subnets. In their work, they 
decouple IP address from TCP transporting layer and enabling 
transport independent fiow in order to solve mobility issues and 
network configuration problems when migrating VMs. This 
approach describes the requirements and the mechanisms used. 





























Figure 3: Hypervisor monitored Live Migration 
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Figure 4: VM networking scheme 


However, it misses the description of live migration scenarios 
that still remain unclear in their prototype. 

In live migration processes any of the VM or the network 
nodes knows the exact time of migration and the destination 
host. VM cant discover that the hypervisor on which it attached 
to the network has been changed after the migration. Therefore, 
the mobility problem in live migration can be easily resolved 
by letting the hypervisor do this work. 

B. Basic concepts for MIP enabled live migration 

The basic components in our conceptual architecture are the 
customer or the client, the home agent and the home node, 
the foreign agent, the Hypervisor (Qemu in our case) and 
the correspondent node. We keep the basis component names 
as in traditional Mobile IP protocol (MIP). Fig. shows the 
four steps for enabling hypervisor controlled mobile IP for 
live migration. Firstly, an administrator located at the Home 
network executes the live migration process. Secondly, a secure 
Tunnel based on SSH security protocol is created between the 
home agent and the remote hypervisor on which the migrated 
VM is running. Thirdly, the hypervisor software redirects the 
incoming traffic to the migrated VM. Fourthly, this latter keeps 
alive its session without any interruption. 

C. Networking System Design 

In our basic architecture several virtualization tools were 
used in order to implement network functions and run them 
on a virtualized environment. Fig. showed our networking 



Figure 5: Live Migration with shared storage 


2-Virtual machine context transfer 



scheme which relies on QEMU/KVM assisted by Libvirt 
daemon. Those tools manage the running VMs and their 
issues. The networking scheme used with virtual managers is 
based on Network Address Translation (NAT) for connecting 
virtual machines to the physical host. Each VM has emulated 
interfaces by which they are differentiated 

D. Virtual Machine Migration Process 

Virtual machine migration in our scenario enables open 
session, session moving, content moving and virtual machine 
moving. We differentiate migration into two types: live VM 
migration with shared storage (SAN: storage area network) and 
live VM migration with context transfer. Process migration is 
still an open problem and session migration does not require 
virtual solutions. 

In Pig. we show a simple scenario of live VM migration 
without context transfer. In this scenario, once the virtual 
machines are executed, the two physical machines share the 
same disk image. 

In Pig. we describe a more complex scenario: live mi¬ 
gration of a network function with context transfer. Before the 
migration process, the migrated VM through the hypervisor 
transfers memory pages and the total disk to the remote host. 
After the Disk transfer, the hypervisor transfers the virtual 
machine context (storage, plug-in, packages, libraries, etc.) to 
the remote host. 
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Figure 7: vCDN based NFV solution 



Figure 8: vCDN based SDN solution 


E. Evaluation 

Starting from our previous networking scheme based on 
Qemu and KVM virtualization tools, we enabled the two way 
of live migration either with shared disk images (single SAN 
or domain) or with context transfer (multiple SANs or multi 
Domain) . 

We have focused on network issues that we have faced in 
live migration in order to more understand the context transfer 
between the interacted physical and virtual machines. Some 
system and networking issues are evaluated in this paper. We 
reported that live migration of running virtual machines needed 
much more requirements in several migrating components 
such as CPU state, storage content, network connections and 
memory content as it showed in Tab. 

We evaluated some experiment measurements basically re¬ 
lated to network parameters such as: link speed that must be 
higher than 1 Gbit/s, live migration time that is too short with 
shred storage 4s but without session interruption in the case 
that we moved streaming point from one host to another. 

V. CDN USE Case 

A. CDN 

Content Distribution Network (CDN) is a large distributed 
network deployed worldwide in order to push content on the 
edge of the network. It was the solution of two major problems 
that decreased the performance of the network: congestion 
within the core network and overload at the origin server. It 
is often used to host static content so that they will be close 
to the end user when he made his request for online video 
streaming. Static contents are: images, video, CSS file, scripts 
files. CDN customers are mainly the Internet Service Providers 


TABLE I: LIVE MIGRATION REQUIREMENTS 
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pages 
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Storage 

Same 
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Transferred, 
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for high 
link speed 
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session 
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memory 

pages 


(ISPs) and Over the Top (OTT) service providers like YouTube 
and Netfiix. All those customers want to push their content to 
the edge servers over IP technology. A CDN is looking to build 
a large farm of cache servers deployed worldwide in order to 
reduce the load on the OTT servers and provide the following 
services: Storage (popularity based). Management of cached 
content (i.e. cache management like replication, replacement, 
and placement). Distribution of content among cache servers 
(load balancing). Content Streaming, Fault tolerance (i.e. Re¬ 
dundancy), and Network performance (reduce load). 

B. Virtualization of CDN 

CDN federation can be proposed to improve performance. 
Virtual CDN, vCDN is a virtual solution for enhancing the 
utilization of the current CDNs. 

Most of the internet traffic is delivered through commer¬ 
cial CDN by allocating some cache servers to distribute the 
content to end users. We introduced that this process can 
be enhanced by three ways of virtualization: classical vCDN 
based Cloud brokers, vCDN based NFV solution, vCDN based 
SDN solution and vCDN based NFV and SDN. Among those 
virtualization ways, we have proposed three new solutions: 

1) vCDN based NFV. 

2) vCDN based SDN. 

3) vCDN using the merge of NFV & SDN. 

Fig. |7] shows our vCDN based NFV solution. Different 
services issued from unique or multiple service providers can 
hence be deployed on virtual machines running as a streaming 
serves on physical servers (Common-Off-The-Shelf (COTS) 
Sever). End users requests are forwarded to the closest vCDN 
virtual machine. Several network issues must be resolved to 
complete NFV and network requirements such as mobility, 
QoS and QoE. 

In Fig. we describe our second architecture using SDN. 
In this approach SDN is only used as a traffic engineering or 
a monitor of edge servers. Through the control protocol Open- 
Flow (OF) enabled in OpenFlow Switches (OFS), vCDN based 
SDN can redirect users request to the closest surrogate server 
and enhance then the overall QoS. The Network Operation 
System (NOS) hosts the SDN controller to communicate with 
the forwarding layer via OF protocol. Several control modules 
and applications can be added to our SDN paradigm to enhance 
the control plane. 









Our third solution: vCDN based NFV and SDN merges the 
two former solutions to virtualize and program CDN software. 
We keep the design of vCDN based NFV solution and we add 
the SDN paradigm to control the virtual edge servers (vCDN) 
based on NFV solutions rather than dedicated hardware. 

VI. Conclusion 

In this paper, we highlighted the impacts of main network 
parameters in the live migration process. We proposed some 
network virtualization concepts like SDN, NFV and Open- 
Stack. We detailed network issues of VM migration in different 
scenarios. We concluded that NF context should be transferred 
for full VNF migration and open session. Virtualized network 
functions can be dynamically instantiated and migrated in a 
virtual environment but they need for high requirements to 
achieve the full context transfer and full virtualization. We 
explained also a use case of vCDN for multimedia service 
objective. 
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